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BACKGROUND 

Decreasing  military  budgets,  reduced  training  infrastructure,  and  increasing  complexity  of  weapon  systems  and 
missions  mandate  the  exploitation  of  innovative  advanced  training  technologies.  In  recent  years,  training  developers 
have  recognized  the  potential  of  virtual  reality  (VR),  often  called  virtual  environments  (VE),  as  a flexible  and  effective 
training  medium,  A prime  candidate  area  for  examining  the  effectiveness  and  usability  of  VR  systems  is  the  training 
of  the  submarine  surfaced  ship  handling  task.  Although  land-based  simulator  facilities  currently  exist  for  training 
Submarine  Piloting  and  Navigation  teams,  these  systems  do  not  provide  detailed  harbor  and  channel  ship  handling  training 
for  the  Officer  of  the  Deck  (OOD).  OOD  training  is  primarily  obtained  from  on-the-job  experience,  which  is  adversely 
impacted  by  the  operational  constraints  of  the  Submarine  Force,  and  the  limited  surfaced  steaming  lime  of  submarines. 
Training  that  will  expose  junior  officers  to  a variety  of  geographical  and  environmental  conditions  is  very  limited  since 
most  Commanding  Officers  place  their  most  experienced  OODs  on  watch  during  these  challenging  evolutions.  Therefore, 
an  alternative,  high-fidelity,  simulation-based  training  capability  is  needed.  A VR-based  simulation  may  provide  this 
necessary  capability  if  it  is  both  effective  and  user-friendly. 

OBJECTIVE  OF  VESUB 

The  goal  of  the  Virtual  Environment  for  Submarine  OOD  Ship  Handling  (VESUB)  project  is  threefold:  (1)  to  develop, 
demonstrate,  and  evaluate  the  training  potential  of  a stand-alone  virtual  reality-based  system  for  OOD  training;  (2)  to 
integrate  this  system  with  existing  Submarine  Piloting  and  Navigation  (SPAN)  training  simulators;  and  (3)  to  determine  the 
viability  of  VR  technology  as  a training  tool. 

DESCRIPTION  OF  VESUB 

The  VESUB  technology  demonstration  system  uses  a high-resolution  head  mounted  display  (HMD)  to  provide  the  trainee 
with  a simulated  360  degree  representation  of  the  visual  environment  containing  many  of  the  required  cues  associated  with 
harbor  and  channel  navigation  as  well  as  accurate  cultural  features  and  varying  environmental  conditions.  Voice 
recognition  and  synthesis  are  used  to  provide  communications  training.  Visual  scene  rendering,  computation  of  harbor 
currents,  wind  effects,  hydrodynamics  of  own  ship  and  traffic  ships  require  state-of-the-art  hardware  and  software, 
including; 

• Silicon  Graphics  Onyx  Image  Generation  Computer  (Infinite  Reality  configuration) 

• n- Vision  Datavisor  HiRes  HMD  (1280  x 1024  pixels  resolution) 

• Polhemus  3 space  Fastrack  Head  Tracker  (Magnetic) 

• ModelGen2  from  Multigen,  Inc.  (creation  of  models  and  terrain) 

• Vega  Marine  from  Paradigm  Simulation,  Inc.  (marine  visual  effects) 

• HARK,  Speaker-Independent  Voice  Recognition  software. 

Additional  details  on  VESUB  hardware  and  software  may  be  found  in  Hays,  Seamon,  and  Bradley  (1997). 

VESUB  DEVELOPMENT  PHASES 

The  VESUB  project  was  organized  in  three  phases:  1)  requirements  determination;  2)  formative  evaluations;  and  3) 
training  effectiveness  evaluations. 


Paper  presented  at  the  RTO  HFM  Workshop  on  "The  Capability  of  Virtual  Reality  to  Meet  Military 
Requirements”,  held  in  Orlando,  USA,  5-9  December  1997,  and  published  in  RTO  MP-54. 


24-2 


Requirements  Determination  Phase 

The  requirements  determination  phase  was  the  first  year  of  the  project.  During  this  phase,  a simplified  feasibility 
demonstration  system,  developed  under  the  NAWCTSD  exploratory  research  Virtual  Environment  Training 
Technology  project,  was  used  to  elicit  VESUB  system  requirements  from  Navy  subject  matter  experts  (SMEs).  The 
initial  functional  requirements  were  documented  in  a NAWCTSD  special  report  (Tenney,  Briscoe,  Pew,  Bradley, 
Seamon,  & Hays,  1996)  and  used  to  direct  the  contractor  during  the  development  of  the  VESUB  technology 
demonstration  system. 

Additional  VESUB  functional  requirements  and  training  objectives  were  developed  around  the  organizing  concept 
“Seaman’s  Eye,”  defined  by  the  VESUB  research  team  as  follows: 

Seaman’s  Eye:  The  total  situation  awareness  of  the  ship  handling  environment  and  the  ability  to 
safely  maneuver  the  vessel  in  all  conditions. 

The  perceptual  and  cognitive  components  of  “Seaman’s  Eye”  were  identified  and  used  to  help  determine  hardware, 
software,  and  instructional  requirements.  Details  on  how  the  perceptual  components  were  used  to  help  select  VESUB 
hardware  may  be  found  in  Hays,  Castillo,  Bradley,  and  Seaman  (1997). 

Formative  Evaluation  Phase 

Formative  evaluations  serve  as  initial  tests  of  the  configuration  and  instructional  methods  included  in  the  training  system. 
These  evaluations  help  the  training  developer  to  avoid  or  correct  costly  mistakes  before  they  are  finalized  in  the 
configuration  of  the  training  system.  The  software  development  and  hardware  integration  for  VESUB  was  accomplished 
under  contract  by  Nichols/Advanced  Marine.  The  contractor  developed  the  VESUB  system  at  their  offices  near 
Washington,  DC.  The  VESUB  formative  evaluations  were  conducted  on  a duplicate  VESUB  system  in  the  laboratory  at 
NAWCTSD  during  the  second  and  third  years  of  the  project.  Each  time  the  development  contractor  produced  an  improved 
iteration  of  the  software,  it  was  installed  in  the  VESUB  laboratory  and  evaluated  against  the  functional  requirements.  Over 
ten  iterations  of  the  system  have  been  produced  and  evaluated. 

Data  for  the  formative  evaluations  were  collected  from  a wide  range  of  SMEs.  These  included:  full  time  retired  Navy 
submariners:  active  duty  Navy  submarine  SMEs  from  the  fleet  and  training  facilities;  and  Navy  reservists. 
Recommendation  for  system  improvements  were  provided  to  the  software  developer  for  improvement  of  the  next  system 
iteration.  A brief  summary  of  the  results  of  the  formative  evaluations  are  presented  below. 

Training  Effectiveness  Evaluations 

The  training  effectiveness  evaluation  (TEE)  phase  will  include  TEEs  of  the  VESUB  technology  demonstration  system. 
The  first  TEE  will  be  conducted  at  the  Submarine  Training  Facility  in  Norfolk,  Virginia  during  January  1998  and  the 
second  will  be  conducted  at  the  Naval  Submarine  School  in  Groton,  Connecticut  during  March  1998.  The  TEEs  will  use 
actual  Navy  trainees  with  various  levels  of  experience  (novice  to  expert)  to  determine  the  effectiveness  of  the  VESUB 
system  and  also  to  help  determine  how  the  technology  can  be  integrated  into  Navy  training.  During  the  Norfolk  TEE, 
VESUB  will  be  connected  to  the  SPAN  trainer  to  determine  if  the  modem  VESUB  hardware  and  software  can 
communicate  with  the  older  SPAN  system.  If  feasible,  the  future  operational  VESUB  systems  will  be  interfaced  to  SPAN 
trainers  to  provide  interactive  OOD  training  with  other  members  of  the  Navigation  Team.  The  results  of  the  TEEs  and 
other  lessons  learned  during  the  VESUB  project  will  be  documented  for  use  in  the  acquisition  of  the  operational  VESUB 
systems. 

Acquisition  of  Operational  VESUB  Systems 

Current  plans  call  for  the  acquisition  of  operational  VESUB  systems,  beginning  in  1999.  The  systems  will  be  installed  at 
the  Navy’s  five  major  submarine  training  facilities.  The  VESUB  technology  demonstrations  system  will  serve  as  a 
functional  baseline,  which  will  have  to  be  matched  or  improved.  The  operational  systems  will  include  the  functionality  of 
the  demonstration  system  and  improve  on  this  functionality,  based  on  the  results  of  the  TEEs. 
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RESULTS  OF  THE  FORMATIVE  EVALUATIONS 

VESUB  formative  evaluations  focused  on  both  the  functionality  of  the  trainee  interface  (e.g.,  the  fidelity  of  objects  in  the 
visual  scene  or  the  responsiveness  of  the  voice  recognition  system),  and  the  usability  of  the  Instructor/Operator  Station 
(IOS).  The  results  of  the  IOS  usability  analysis  are  presented  in  Hays,  Seamon,  and  Bradley  (1997).  This  report 
documents  twenty-two  areas  where  the  design  of  the  operational  VESUB  system  IOS  can  be  improved. 

Hundreds  of  SME  recommendations  for  improvement  of  the  trainee  interface  were  collected  during  the  formative 
evaluations.  These  recommendations  ranged  from  very  general  (e.g.,  improvements  in  the  representation  of  equipment 
on  the  submarine  bridge)  to  very  specific  (e.g.,  the  logic  supporting  the  “angle  on  the  bow”  display  on  the  IOS).  The 
recommendations  were  provided  to  the  system  developer  and  prioritized  for  inclusion  in  subsequent  software 
iterations.  Only  a limited  number  of  the  recommendations  could  be  included  in  the  demonstration  system  due  to 
budget,  schedule,  and  technology  constraints.  For  example,  in  March  and  April  of  1997,  seventy-three  major  system 
requirements  were  reduced  to  twelve.  Two  examples  of  requirements  that  were  not  implemented  are:  time  varying 
fog  (e.g.,  fog  bank  “rolling”  in  rather  than  instant  reduction  of  visibility)  and  secondary  propulsion  motors  (used  when 
the  rudder  or  main  engine  fails  and  for  docking  maneuvers).  Two  examples  of  requirements  that  were  included  in  the 
system  are:  environment  sounds  (e.g.,  ship  whistles,  bells  on  buoys,  wind,  and  rain);  and  latitude/longitude 

organization  of  the  databases  (rather  than  an  x-y  coordinate  system  as  in  previous  ship  handling  trainers).  An  example 
of  a partially  developed  function,  which  will  require  improvement  in  the  operational  system,  is  collision  detection. 
Finally,  several  unanticipated  requirements  were  recognized  only  after  initial  versions  of  the  software  were  evaluated. 
An  example  of  these  unanticipated  requirements  is  large  area  database  management  (required  because  of  the  need  to 
display  both  close  and  distant  objects  in  the  visual  field).  Details  on  the  trade-offs  made  during  system  development 
will  be  documented  in  a final  “lessons  learned”  report. 

CONCLUSIONS 

The  development  of  a complex  training  system,  like  VESUB,  requires  the  dedicated  efforts  of  a team  of  experts  including: 
computer  scientists,  visual  engineers,  task  SMEs,  and  instructional  psychologists.  It  is  essential  that  a substantial  period  of 
formative  evaluations  be  included  in  the  development  of  any  such  training  system.  It  is  not  possible  to  anticipate  even  a 
fraction  of  the  problems  that  will  be  encountered  during  system  development.  Formative  evaluations  provide  the 
opportunity  to  correct  problems  before  they  are  fully  implemented  in  the  system  and  become  too  costly  to  correct.  As 
training  systems,  VE-based  or  otherwise,  become  more  advanced,  they  also  become  more  complex.  This  complexity 
brings  unanticipated  difficulties,  which  will  require  innovative  and  timely  solutions.  Formative  evaluations  provide  the 
opportunity  for  system  developers  to  apply  these  solutions  at  a stage  where  they  can  have  a positive  benefit  to  training 
system  effectiveness. 
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